Relative optical path phase reconstruction in the correction of misfocused images using fractional powers of the fourier transform

ABSTRACT

Reconstructing relative phase information in a misfocused image correction process for misfocused image data having image amplitude information, but lacking relative phase information. A suitable automated or human-operated process is utilized to generate an appropriate power value of a fractional Fourier transform correction operation. This power value is then used to simultaneously calculate the Fractional Fourier transform correction operation and back-calculate corresponding phase restoration information in an iterative or non-iterative environment. The phase restoration information and fractional Fourier transform correction operation may be applied to the image data to correct a desired level of misfocus in image data obtained from conventional lens systems or other fractional Fourier environments within integrated optics, optical computing, astronomical observation, and particle beam systems such as accelerators and electron microscopes, and may be incorporated into film processing, photo editing software and web sites, cameras, VCRs, video editing, surveillance, and conferencing video systems.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of U.S. application Ser. No.09/512,775 entitled “CORRECTION OF UNFOCUSED LENS EFFECTS VIA FRACTIONALFOURIER TRANSFORM” filed Feb. 25, 2000, now U.S. Pat. No. 6,687,418,which claims benefit of priority of U.S. provisional applications Ser.Nos. 60/121,680 and 60/121,958, each filed on Feb. 25, 1999.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates to optical signal processing, and moreparticularly to the use of fractional Fourier transform properties oflenses to correct the effects of lens misfocus in photographs, video,and other types of captured images.

2. Discussion of the Related Art

A number of references are cited herein; these are provided in anumbered list at the end of the detailed description of the preferredembodiments. These references are cited at various locations throughoutthe specification using a reference number enclosed in square brackets.

The Fourier transforming properties of simple lenses and related opticalelements is well known and heavily used in a branch of engineering knownas Fourier optics [1,2]. Classical Fourier optics [1,2,3,4] utilizelenses or other means to obtain a two-dimensional Fourier transform ofan optical wavefront, thus creating a Fourier plane at a particularspatial location relative to an associated lens. This Fourier planeincludes an amplitude distribution of an original two-dimensionaloptical image, which becomes the two-dimensional Fourier transform ofitself. In the far simpler area of classical geometric optics [1,3],lenses and related objects are used to change the magnification of atwo-dimensional image according to the geometric relationship of theclassical lens-law. It has been shown that between the geometriesrequired for classical Fourier optics and classical geometric optics,the action of a lens or related object acts on the amplitudedistribution of images as the fractional power of the two-dimensionalFourier transform. The fractional power of the fractional Fouriertransform is determined by the focal length characteristics of the lens,and the relative spatial separation between a lens, source image, and anobserved image.

The fractional Fourier transform has been independently discovered onvarious occasions over the years [5,7,8,9,10], and is related to severaltypes of mathematical objects such as the Bargmann transform [8] and theHermite semigroup [13]. As shown in [5], the most general form ofoptical properties of lenses and other related elements [1,2,3] can betransformed into a fractional Fourier transform representation. Thisproperty has apparently been rediscovered some years later and worked onsteadily ever since (see for example [6]), expanding the number ofoptical elements and situations covered. It is important to remark,however, that the lens modeling approach in the latter ongoing series ofpapers view the multiplicative phase term in the true form of thefractional Fourier transform as a problem or annoyance and usually omitit from consideration.

SUMMARY OF THE INVENTION

Correction of the effects of misfocusing in recorded or real-time imagedata may be accomplished using fractional Fourier transform operationsrealized optically, computationally, or electronically. In someembodiments, the invention extends the capabilities of using a power ofthe fractional Fourier transform for correcting misfocused images, tosituations where phase information associated with the original imagemisfocus is unavailable. For example, conventional photographic andelectronic image capture, storage, and production technologies can onlycapture and process image amplitude information—the relative phaseinformation created within the original optical path is lost. As will bedescribed herein, the missing phase information can be reconstructed andused when correcting image misfocus.

In accordance with some embodiments, algebraic group properties of thefractional Fourier transform are used to back-calculate lost originalrelative phase conditions that would have existed if a given specificcorrective operation were to correct a misfocused image. Correctiveiterations can then be made to converge on a corrected focus condition.Simplified numerical calculations of phase reconstructions may beobtained by leveraging additional properties of the fractional Fouriertransform for employing pre-computed phase reconstructions.

Some embodiments use the inherent fractional Fourier transformproperties of lenses or related elements or environments, such ascompound lenses or graded-index materials, to correct unfocused effectsof various types of captured images. Use of the algebraic unitary groupproperty of the fractional Fourier transform allows for a simplecharacterization of the exact inverse operation for the initialmisfocus.

One aspect of the present invention reconstructs relative phaseinformation affiliated with the original misfocused optical path in thecorrection of misfocused images.

Another aspect of the present invention provides for the calculation ofan associated reconstruction of relative phase information, which wouldbe accurate if an associated trial fractional Fourier transform powerwere the one to correct the focus of the original misfocused image.

Another aspect of the present invention provides for simplifiedcalculation of the phase reconstruction information using algebraicgroup and antisymmetry properties of the fractional Fourier transformoperator.

Still yet another aspect of the present invention provides forsimplified calculation of the phase reconstruction information usingmodified calculations of fractional powers of the Fourier transform.

Another aspect of the present invention provides for simplifiedcalculation of the phase reconstruction information by rearranging termsin the calculation of a fractional Fourier transform operator.

Yet another aspect of the present invention provides for simplifiedcalculation of the phase reconstruction information using a partition ofterms in the calculation of a fractional Fourier transform operator.

Still yet another aspect of the present invention provides for relevantfractional Fourier transform operations to be accomplished directly orapproximately by means of optical components, numerical computer,digital signal processing, or other signal processing methods orenvironments.

Yet another aspect of the present invention provides approximationmethods which leverage Hermite function expansions which can beadvantageous in that the orthogonal Hermite functions diagonalize theFourier transform and fractional Fourier transform, yielding thefollowing two-fold result. First, throughout the entire optical system,the amplitude and phase affairs of each Hermite function are completelyindependent of those of the other Hermite functions. Second, the Hermitefunction expansion of a desired transfer function will naturally havecoefficients that eventually go to zero, meaning that to obtain anarbitrary degree of approximation in some situations, only a manageablenumber of Hermite functions need be considered.

Another aspect of the present invention allows the power of thefractional Fourier transform to be determined by automatic methods.These automatic methods may include edge detection elements andprovisions for partial or complete overriding by a human operator.

Still yet another aspect of the present invention provides for thefraction Fourier transform power to be determined entirely by a humanoperator.

Another aspect of the invention provides for pre-computed values ofphase reconstructions corresponding to pre-computed powers of fractionalFourier transform to be composed to create phase reconstructionscorresponding to other powers of the fractional Fourier transform.

Another aspect of the present invention provides for at least onepre-computed power of the fractional Fourier transform to be used incomputing or approximating higher powers of the fractional Fouriertransform.

Yet another aspect of the present invention provides for pre-computedvalues of phase reconstructions corresponding to powers of fractionalFourier transform, wherein the powers are related by roots of the number2 or in other ways to leverage fractional expansion.

Still yet another aspect of the present invention provides for composedphase reconstructions that are realized in correspondence to binaryrepresentations of fractions.

Yet another aspect of the present invention provides for combiningnumerical correction of video camera lens misfocus with videodecompression algorithms to increase performance and reduce requiredmisfocus-correction computations.

The present invention enables the recovery of misfocused images obtainedfrom photographs, video, movies, and other types of captured images.Because a high quality lens or lens system operates on the amplitudedistribution of the source image as a two-dimensional fractional Fouriertransform, the algebraic unitary group property of the fractionalFourier transform allows for the exact calculation of the inverseoperation for initial lens misfocus. Additional mathematical propertiesof the fractional Fourier transform allow for different methods ofapproximation meaningful in the economic embodiments of the invention.The system and method provided herein enable economic and wide-rangingimplementation for after-capture correction of image misfocus, forexample.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features and advantages of the presentinvention will become more apparent upon consideration of the followingdescription of preferred embodiments taken in conjunction with theaccompanying drawing figures, wherein:

FIG. 1 is a block diagram showing a general lens arrangement andassociated image observation entity capable of classical geometricoptics, classical Fourier optics, and fractional Fourier transformoptics;

FIG. 2 is a block diagram showing an exemplary approach for automatedadjustment of fractional Fourier transform parameters for maximizing thesharp edge content of a corrected image, in accordance with oneembodiment of the present invention;

FIG. 3 is a block diagram showing a typical approach for adjusting thefractional Fourier transform parameters to maximize misfocus correctionof an image, in accordance with one embodiment of the present invention;

FIG. 4 is a diagram showing a generalized optical environment forimplementing image correction in accordance with the present invention;

FIG. 5 is a diagram showing focused and unfocused image planes inrelationship to the optical environment depicted in FIG. 4;

FIG. 6 is a block diagram showing an exemplary image misfocus correctionprocess that also provides phase corrections;

FIG. 7 is a diagram showing a more detailed view of the focused andunfocused image planes shown in FIG. 5;

FIG. 8 is a diagram showing typical phase shifts involved in the focusedand unfocused image planes depicted in FIG. 5;

FIG. 9 shows techniques for computing phase correction determined by thefractional Fourier transform applied to a misfocused image; and

FIG. 10 is a block diagram showing an exemplary image misfocuscorrection process that also provides for phase correction, inaccordance with an alternative embodiment of the invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

In the following description, reference is made to the accompanyingdrawing figures which form a part hereof, and which show by way ofillustration specific embodiments of the invention. It is to beunderstood by those of ordinary skill in this technological field thatother embodiments may be utilized, and structural, electrical, optical,as well as procedural changes may be made without departing from thescope of the present invention.

As used herein, the term “image” refers to both still-images (such asphotographs, video frames, video stills, movie frames, and the like) andmoving images (such as motion video and movies). Many embodiments of thepresent invention are directed to processing recorded or real-time imagedata provided by an exogenous system, means, or method. Presented imagedata may be obtained from a suitable electronic display such as an LCDpanel, CRT, LED array, films, slides, illuminated photographs, and thelike. Alternatively or additionally, the presented image data may be theoutput of some exogenous system such as an optical computer orintegrated optics device, to name a few. The presented image data willalso be referred to herein as the image source.

If desired, the system may output generated image data having someamount of misfocus correction. Generated image data may be presented toa person, sensor (such as a CCD image sensor, photo-transistor array,for example), or some exogenous system such as an optical computer,integrated optics device, and the like. The entity receiving generatedimage data will be referred to as an observer, image observation entity,or observation entity.

Reference will first be made to FIG. 3 which shows a general approachfor adjusting the fractional Fourier transform parameters to maximizethe correction of misfocus in an image. Details regarding the use of afractional Fourier transform (with adjusted parameters of exponentialpower and scale) to correct image misfocus will be later described withregard to FIGS. 1 and 2.

Original visual scene 301 (or other image source) may be observed byoptical system 302 (such as a camera and lens arrangement) to produceoriginal image data 303 a. In accordance with some embodiments, opticalsystem 302 may be limited, misadjusted, or otherwise defective to theextent that it introduces a degree of misfocus into the imagerepresented by the image data 303 a. It is typically not possible orpractical to correct this misfocus effect at optical system 302 toproduce a better focused version of original image data 303 a.Misfocused original image data 303 a may be stored over time ortransported over distance. During such a process, the original imagedata may be transmitted, converted, compressed, decompressed, orotherwise degraded, resulting in an identical or perturbed version oforiginal image data 303 b. It is this perturbed version of the originalimage data that may be improved using the misfocus correction techniquesdisclosed herein. Original and perturbed image data 303 a, 303 b may bein the form of an electronic signal, data file, photography paper, orother image form.

Original image data 303 b may be manipulated numerically, optically, orby other means to perform a fractional Fourier transform operation 304on the original image data to produce resulting (modified) image data305. The parameters of exponential power and scale factors of thefractional Fourier transform operation 304 may be adjusted 310 over somerange of values, and each parameter setting within this range may resultin a different version of resulting image data 305. As the level ofmisfocus correction progresses, the resulting image data 305 will appearmore in focus. The improvement in focus will generally be obvious to anattentive human visual observer, and will typically be signified by anincrease in image sharpness, particularly at any edges that appear inthe image. Thus a human operator, a machine control system, or acombination of each can compare a sequence of resulting images createdby previously selected parameter settings 310, and try a new parametersetting for a yet another potential improvement.

For a human operator, this typically would be a matter of adjusting acontrol and comparing images side by side (facilitated by non-humanmemory) or, as in the case of a microscope or telescope, by comparisonfacilitated purely with human memory. For a machine, a systematiciterative or other feedback control scheme would typically be used. InFIG. 3, each of these image adjustments is generalized by the steps andelements suggested by interconnected elements 306–309, although othersystems or methods accomplishing the same goal with different internalstructure (for example, an analog electronic circuit, optical materials,or chemical process) are provided for and anticipated by the presentinvention. For the illustrative general case of FIG. 3, resulting imagedata 305 for selected parameter settings 310 may be stored in human,machine, or photographic memory 306, along with the associated parametersettings, and compared 307 for the quality of image focus. Based onthese comparisons, subsequent high level actions 308 may be chosen.

High level actions 308 typically require translation into new parametervalues and their realization, which may be provided by parametercalculation and control 309. This process may continue for some intervalof time, some number of resulting images 305, or some chosen orpre-determined maximum level of improvement. One or more “best choice”resulting image data set or sets 305 may then be identified as theresult of the action and processes depicted in this figure.

With this high level description having been established, attention isnow directed to details of the properties and use of a fractionalFourier transform (with adjusted parameters of exponential power andscale) to correct misfocus in an image and maximize correction ofmisfocus. This aspect of the present invention will be described withregard to FIG. 1.

FIG. 1 is a block diagram showing image source 101, lens 102, and imageobservation entity 103. The term “lens” is used herein for convenience,but it is to be understood that the image misfocus correction techniquesdisclosed herein apply equally to lens systems and other similar opticalenvironments. The image observation entity may be configured withclassical geometric optics, classical Fourier optics, or fractionalFourier transform optics. The particular class of optics (geometric,Fourier, or fractional Fourier) implemented in a certain application maybe determined using any of the following:

-   -   separation distances 111 and 112;    -   the “focal length” parameter “f” of lens 102;    -   the type of image source (lit object, projection screen, etc.)        in as far as whether a plane or spherical wave is emitted.

As is well known, in situations where the source image is a lit objectand where distance 111, which shall be called “a,” and distance 112,which shall be called “b,” fall

$\begin{matrix}{\frac{1}{f} = {\frac{1}{a} + \frac{1}{b}}} & (1)\end{matrix}$into the lens-law relationship, may be determined by the focal length f:which gives the geometric optics case. In this case, observed image 103is a vertically and horizontally inverted version of the original imagefrom source 101, scaled in size by a magnification factor m given by:

$\begin{matrix}{m = \frac{b}{a}} & (2)\end{matrix}$

As previously noted, the Fourier transforming properties of simplelenses and related optical elements is also well known in the field ofFourier optics [2,3]. Classical Fourier optics [2,3,4,5] involve the useof a lens, for example, to take a first two-dimensional Fouriertransform of an optical wavefront, thus creating a Fourier plane at aparticular spatial location such that the amplitude distribution of anoriginal two-dimensional optical image becomes the two-dimensionalFourier transform of itself. In the arrangement depicted in FIG. 1, witha lit object serving as source image 101, the Fourier optics case may beobtained when a=b=f.

As described in [5], for cases where a, b, and f do not satisfy the lenslaw of the Fourier optics condition above, the amplitude distribution ofsource image 101, as observed at observation entity 103, experiences theaction of a non-integer power of the Fourier transform operator. Asdescribed in [5], this power, which shall be called α, varies between 0and 2 and is determined by an Arc-Cosine function dependent on the lensfocal length and the distances between the lens, image source, and imageobserver; specifically:

$\begin{matrix}{\alpha = {\frac{2}{\pi}{\arccos\left\lbrack {{{sgn}\left( {f - a} \right)}\frac{\sqrt{\left( {f - a} \right)\left( {f - b} \right)}}{f}} \right\rbrack}}} & (3)\end{matrix}$for cases where (f−a) and (f−b) share the same sign. There are othercases which can be solved from the more primitive equations in [5] (atthe bottom of pages ThE4-3 and ThE4-1). Note simple substitutions showthat the lens law relationship among a, b, and f indeed give a power of2, and that the Fourier optics condition of a=b=f give the power of 1,as required.

The fractional Fourier transform properties of lenses typically causecomplex but predictable phase and scale variations. These variations maybe expressed in terms of Hermite functions, as presented shortly, but itis understood that other representations of the effects, such asclosed-form integral representations given in [5], are also possible anduseful.

Several methods can be used to construct the fractional Fouriertransform, but to begin it is illustrative to use the orthogonal Hermitefunctions, which as eigenfunctions diagonalize the Fourier transform[17]. Consider the Hermite function [16] expansion [17, and morerecently, 18] of the two dimensional image amplitude distributionfunction. In one dimension, a bounded (i.e., non-infinite) function k(x)can be represented as an infinite sum of Hermite functions {h_(n)(x)}as:

$\begin{matrix}{{k(x)} = {\sum\limits_{n = 0}^{\infty}{a_{n}{h_{n}(x)}}}} & (4)\end{matrix}$

Since the function is bounded, the coefficients {a_(n)} eventuallybecome smaller and converge to zero. An image may be treated as a twodimensional entity (for example, a two-dimensional array of pixels), orit can be the amplitude variation of a translucent plate; in eithercase, the function may be represented in a two-dimensional expansionsuch as:

$\begin{matrix}{{k\left( {x_{1},x_{2}} \right)} = {\sum\limits_{m = 0}^{\infty}{\sum\limits_{n = 0}^{\infty}{a_{n,m}{h_{n}\left( x_{1} \right)}{h_{m}\left( x_{2} \right)}}}}} & (5)\end{matrix}$For simplicity, the one dimensional case may be considered. The Fouriertransform action on Hermite expansion of the function k(x) with seriescoefficients {a_(n)} is given by [16]:

$\begin{matrix}{{F\left\lbrack {k(x)} \right\rbrack} = {\sum\limits_{n = 0}^{\infty}{\left( {- {\mathbb{i}}} \right)^{n}a_{n}{h_{n}(x)}}}} & (6)\end{matrix}$Because of the diagonal eigenfunction structure, fractional powers ofthe Fourier transform operator may be obtained by taking the fractionalpower of each eigenfunction coefficient. The eigenfunction coefficientshere are (−i)^(n). Complex branching artifact ambiguities that arisefrom taking the roots of complex numbers can be avoided by writing (−i)as:e^(−iπ/2)  (7)Thus for a given power α, the fractional Fourier transform of theHermite expansion of the function k(x) with series coefficients {a_(n)}can be given by [5]:

$\begin{matrix}{{F^{\alpha}\left\lbrack {k(x)} \right\rbrack} = {\sum\limits_{n = 0}^{\infty}{{\mathbb{e}}^{{- {\mathbb{i}}}\; n\;\pi\;{\alpha/2}}a_{n}{h_{n}(x)}}}} & (8)\end{matrix}$Note when α=1, the result is the traditional Fourier transform above,and when α=2, the result may be expressed as:

$\begin{matrix}\begin{matrix}{{F^{2}\left\lbrack {k(x)} \right\rbrack} = {{\sum\limits_{n = 0}^{\infty}{{\mathbb{e}}^{{- {\mathbb{i}}}\; n\;\pi}a_{n}{h_{n}(x)}}} = {\sum\limits_{n = 0}^{\infty}{\left( {- 1} \right)^{n}a_{n}{h_{n}(x)}}}}} \\{= {{\sum\limits_{n = 0}^{\infty}{a_{n}{h_{n}\left( {- x} \right)}}} = {k\left( {- x} \right)}}}\end{matrix} & (9)\end{matrix}$due to the odd and even symmetry, respectively, of the odd and evenHermite functions. This is the case for the horizontally and verticallyinverted image associated with the lens law of geometric optics,although here the scale factors determining the magnification factorhave been normalized out.

More generally, as the power α varies (via the Arccosine relationshipdepending on the separation distance), the phase angle of the n^(th)coefficient of the Hermite expansion varies according to therelationship shown above and the scale factor may vary as well [5]. Forimages, all of the above occurs in the same manner but in two dimensions[5].

Through use of the Mehler kernel [16], the above expansion may berepresented in closed form as [5]:

$\begin{matrix}{{F^{\alpha}\left\lbrack {k(x)} \right\rbrack} = {\sqrt{\frac{{\mathbb{e}}^{{- \pi}\;\alpha\;{{\mathbb{i}}/2}}}{{\mathbb{i}}\mspace{11mu}{\sin\left( {\pi\;{\alpha/2}} \right)}}}{\int_{- \infty}^{\infty}{{k(x)}{\mathbb{e}}^{2^{\pi}{{\mathbb{i}}\;\lbrack{{{(\frac{x^{2} + y^{2}}{2})}{\cot{(\frac{\pi\;\alpha}{2})}}} - {{xycsc}{(\frac{\pi\;\alpha}{2})}}}\rbrack}}{\mathbb{d}x}}}}} & (10)\end{matrix}$Note in [5] that the factor of i multiplying the sin function under theradical has been erroneously omitted. Clearly, both the Hermite andintegral representations are periodic in α with period four. Further, itcan be seen from either representation that:F ^(2±ε) [k(x)]=F ² F ^(±ε) [k(x)]=F ^(±ε) F ² [k(x)]=F ^(±ε)[k(−x)]  (11)which illustrates an aspect of the invention as the effect ε will be thedegree of misfocus introduced by the misfocused lens, while the Fouriertransform raised to the second power represents the lens-law opticscase. In particular, the group property makes it possible to calculatethe inverse operation to the effect induced on a record image by amisfocused lens in terms of explicit mathematical operations that can berealized either computationally, by means of an optical system, or both.Specifically, because the group has period 4, it follows that F⁻²=F²;thus:(F ^(2±ε) [k(x)])⁻¹ =F ⁻² F ^(∓ε) [k(x)]=F ² F ^(∓ε) [k(x)]=F ^(∓ε) F ²[k(x)]=F ^(∓ε) [k(−x)]  (12)Thus, one aspect of the invention provides image misfocus correction,where the misfocused image had been created by a quality thoughmisfocused lens or lens-system. This misfocus can be corrected byapplying a fractional Fourier transform operation; and morespecifically, if the lens is misfocused by an amount corresponding tothe fractional Fourier transform of power ε, the misfocus may becorrected by applying a fractional Fourier transform operation of power−ε.

It is understood that in some types of situations, spatial scale factorsof the image may need to be adjusted in conjunction with the fractionalFourier transform power. For small variations of the fractional Fouriertransform power associated with a slight misfocus, this is unlikely tobe necessary. However, should spatial scaling need to be made, variousoptical and signal processing methods well known to those skilled in theart can be incorporated. In the case of pixilated images (imagesgenerated by digital cameras, for example) or lined-images (generated byvideo-based systems, for example), numerical signal processingoperations may require standard resampling (interpolation and/ordecimation) as is well known to those familiar with standard signalprocessing techniques.

It is likely that the value of power ∈ is unknown a priori. In thisparticular circumstance, the power of the correcting fractional Fouriertransform operation may be varied until the resulting image is optimallysharpened. This variation could be done by human interaction, as withconventional human interaction of lens focus adjustments on a camera ormicroscope, for example.

If desired, this variation could be automated using, for example, somesort of detector in an overall negative feedback situation. Inparticular, it is noted that a function with sharp edges are obtainedonly when its contributing, smoothly-shaped basis functions have veryparticular phase adjustments, and perturbations of these phaserelationships rapidly smooth and disperse the sharpness of the edges.Most natural images contain some non-zero content of sharp edges, andfurther it would be quite unlikely that a naturally occurring, smoothgradient would tighten into a sharp edge under the action of thefractional Fourier transform because of the extraordinary basis phaserelationships required. This suggests that a spatial high-pass filter,differentiator, or other edge detector could be used as part of thesensor makeup. In particular, an automatically adjusting system may beconfigured to adjust the fractional Fourier transform power to maximizethe sharp edge content of the resulting correcting image. If desired,such a system may also be configured with human override capabilities tofacilitate pathological image situations, for example.

FIG. 2 shows an automated approach for adjusting the fractional Fouriertransform parameters of exponential power and scale factor to maximizethe sharp edge content of the resulting correcting image. In thisfigure, original image data 201 is presented to an adjustable fractionalFourier transform element 202, which may be realized physically viaoptical processes or numerically (using an image processing orcomputation system, for example). The power and scale factors of thefractional Fourier transform may be set and adjusted 203 as necessaryunder the control of a step direction and size control element 204.

Typically, this element would initially set the power to the ideal valueof zero (making the resulting image data 205 equivalent to the originalimage data 201) or two (making the resulting image data 205 equivalentto an inverted image of original image data 201), and then deviateslightly in either direction from this initial value. The resultingimage data 205 may be presented to edge detector 206 which identifiesedges, via differentiation or other means, whose sharpness passes aspecified fixed or adaptive threshold. The identified edge informationmay be passed to an edge percentage tally element 207, which transformsthis information into a scalar-valued measure of the relative degree ofthe amount of edges, using this as a measure of image sharpness.

The scalar measure value for each fractional Fourier transform power maybe stored in memory 208, and presented to step direction and sizecontrol element 204. The step direction and size control elementcompares this value with the information stored in memory 208 andadjusts the choice of the next value of fractional Fourier transformpower accordingly. In some implementations, the step direction and sizecontrol element may also control edge detection parameters, such as thesharpness threshold of edge detector element 207. When the optimaladjustment is determined, image data 205 associated with the optimalfractional Fourier transform power is designated as the corrected image.

It is understood that the above system amounts to a negative-feedbackcontrol or adaptive control system with a fixed or adaptive observer. Assuch, it is understood that alternate means of realizing this automatedadjustment can be applied by those skilled in the art. It is also clearto one skilled in the art that various means of interactive humanintervention may be introduced into this automatic system to handleproblem cases or as a full replacement for the automated system.

In general, the corrective fractional Fourier transform operation can beaccomplished by any one or combination of optical, numerical computer,or digital signal processing methods as known to those familiar with theart, recognizing yet other methods may also be possible. Optical methodsmay give effectively exact implementations of the fractional Fouriertransforms, or in some instances, approximate implementations of thetransforms. For a pixilated image, numerical or other signal processingmethods may give exact implementations through use of the discreteversion of the fractional Fourier transform [10].

Additional computation methods that are possible include one or more of:

-   -   dropping the leading scalar complex-valued phase term (which        typically has little or no effect on the image);    -   decomposing the fractional Fourier transform as a        pre-multiplication by a “phase chirp” e^(icz2), taking a        conventional Fourier transform with appropriately scaled        variables, and multiplying the result by another “phase chirp;”        and    -   changing coordinate systems to Wigner form:

$\begin{matrix}\left\{ {\frac{\left( {x + y} \right)}{w},\frac{\left( {x - y} \right)}{w}} \right\} & (13)\end{matrix}$If desired, any of these just-described computation methods can be usedwith the approximating methods described below.

Other embodiments provide approximation methods for realizing thecorrective fractional Fourier transform operation. For a non-pixilatedimage, numerical or other signal processing methods can giveapproximations through:

-   -   finite-order discrete approximations of the integral        representation;    -   finite-term discrete approximations by means of the Hermite        expansion representation; and    -   the discrete version of the fractional Fourier transform [10].

Classical approximation methods [11,12] may be used in the latter twocases to accommodate particular engineering, quality, or costconsiderations.

In the case of Hermite expansions, the number of included terms may bedetermined by analyzing the Hermite expansion of the image data, shouldthis be tractable. In general, there will be some value in situationswhere the Hermite function expansion of the image looses amplitude asthe order of the Hermite functions increases. Hermite function orderswith zero or near-zero amplitudes may be neglected entirely from thefractional Fourier computation due to the eigenfunction role of theHermite functions in the fractional Fourier transform operator.

One method for realizing finite-order discrete approximations of theintegral representation would be to employ a localized perturbation orTaylor series expansion of the integral representation. In principal,this approach typically requires some mathematical care in order for theoperator to act as a reflection operator (i.e., inversion of eachhorizontal direction and vertical direction as with the lens law) sincethe kernel behaves as a generalized function (delta function), and hencethe integral representation of the fractional Fourier transform operatorresembles a singular integral.

In a compound lens or other composite optical system, the reflectionoperator may be replaced with the identity operator, which also involvesvirtually identical delta functions and singular integrals as is knownto those familiar in the art. However, this situation is fairly easy tohandle as a first or second-order Taylor series expansion. The requiredfirst, second, and any higher-order derivatives of the fractionalFourier transform integral operator are readily and accurately obtainedsymbolically using available mathematical software programs, such asMathematica or MathLab, with symbolic differential calculuscapabilities. In most cases, the zero-order term in the expansion willbe the simple reflection or identity operator. The resulting expansionmay then be numerically approximated using conventional methods.

Another method for realizing finite-order discrete approximations of theintegral representation would be to employ the infinitesimal generatorof the fractional Fourier transform, that is, the derivative of thefractional Fourier transform with respect to the power of the transform.This is readily computed by differentiating the Hermite functionexpansion of the fractional Fourier transform, and use of the derivativerule for Hermite functions. Depending on the representation used[5,14,15], the infinitesimal generator may be formed as a linearcombination of the Hamiltonian operator H and the identity operator I;for the form of the integral representation used earlier, this would be:

$\begin{matrix}{\frac{{\mathbb{i}}\;\pi}{4}\left( {H + I} \right)} & (14)\end{matrix}$where and the identity operator I simply reproduces the originalfunction, and

$\begin{matrix}{H = {\frac{\partial^{2}}{\partial x^{2}} - x^{2}}} & (15)\end{matrix}$The role of the infinitesimal generator, which can be denoted as A, isto represent an operator group in exponential form, a particular exampleis:F ^(α) =e ^(αA)  (16)For small values of A, one can then approximate e^(αA) as I+(αA), sousing the fact [12] from before (repeated here):(F ^(2±ε) [k(x)])⁻¹ =F ⁻² F ^(∓ε) [k(x)]=F ² F ^(∓ε) [k(x)]=F ^(∓ε) F ²[k(x)]=F ^(∓ε) [k(−x)]  (17)one can then approximate F^(∈) as

$\begin{matrix}{F^{ɛ} = {{I + \left( {ɛ\; A} \right)} = {I + {ɛ\frac{{\mathbb{i}}\;\pi}{4}\left( {\frac{\partial^{2}}{\partial x^{2}} - x^{2} + I} \right)}}}} & (18)\end{matrix}$These operations can be readily applied to images using conventionalimage processing methods.

For non-pixilated images, the original source image can be approximatedby two-dimensional sampling, and the resulting pixilated image can thenbe subjected to the discrete version of the fractional Fourier transform[10].

In cases where the discrete version of the fractional Fourier transform[10] is implemented, the transform may be approximated. The discreterepresentation can, for example, be a three-dimensional matrix (tensor)operator. Alternatively, pairs of standard two-dimensional matrices, onefor each dimension of the image, can be used. As with the continuouscase, various types of analogous series approximations, such as thoseabove, can be used.

Finally, it is noted that because of the commutative group property ofthe fractional Fourier transform, the matrix/tensor representations, orin some realizations even the integrals cited above may be approximatedby pre-computing one or more fixed step sizes and applying theserespectively, iteratively, or in mixed succession to the image data.

One exemplary embodiment utilizing a pre-computation technique may bewhere the fractional Fourier transform represents pre-computed, positiveand negative values of a small power, for example 0.01. Negative powerdeviations of increasing power can be had by iteratively applying thepre-computed −0.01 power fractional Fourier transform; for example, thepower −0.05 would be realized by applying the pre-computed −0.01 powerfractional Fourier transform five times. In some cases of adaptivesystem realizations, it may be advantageous to discard some of theresulting image data from previous power calculations. This may beaccomplished by backing up to a slightly less negative power by applyingthe +0.01 power fractional Fourier transform to a last stored, resultingimage.

As a second example of this pre-computation method, pre-computedfractional Fourier transform powers obtained from values of the series2^(1/N) and 2^(−1/N) may be stored or otherwise made available, forexample:{F^(±1/1024),F^(±1/512),F^(±1/256),F^(±1/128),F^(±1/64), . . . }  (19)Then, for example, the power 11/1024 can be realized by operating on theimage data withF^(1/1024)F^(1/256)F^(1/128)  (20)where the pre-computed operators used are determined by thebinary-decomposition of the power with respect to the smallest powervalue (here, the smallest value is 1/1024 and the binary decompositionof 11/1024 is 1/1024+ 1/256+ 1/128, following from the fact that11=8+2+1). Such an approach allows, for example, N steps of resolutionto be obtained from a maximum of log₂N compositions of log₂Npre-computed values.

It is noted that any of the aforementioned systems and methods may beadapted for use on portions of an image rather than the entire image.This permits corrections of localized optical aberrations. Incomplicated optical aberration situations, more than one portion of animage may be processed in this manner, with differing correctiveoperations made for each portion of the image.

Finally, it is noted that the systems and methods described herein mayalso be applied to conventional lens-based optical image processingsystems, to systems with other types of elements obeying fractionalFourier optical models, as well as to widely ranging environments suchas integrated optics, optical computing systems, particle beam systems,electron microscopes, radiation accelerators, and astronomicalobservation methods, among others.

Commercial products and services application are widespread. Forexample, the present invention may be incorporated into film processingmachines, desktop photo editing software, photo editing web sites, VCRs,camcorders, desktop video editing systems, video surveillance systems,video conferencing systems, as well as in other types of products andservice facilities. Four exemplary consumer-based applications are nowconsidered.

1. One particular consumer-based application is in the correction ofcamera misfocus in chemical or digital photography. Here the inventionmay be used to process the image optically or digitally, or somecombination thereof, to correct the misfocus effect and create animproved image which is then used to produce a new chemical photographor digital image data file. In this application area, the invention canbe incorporated into film processing machines, desktop photo editingsoftware, photo editing web sites, and the like.

2. Another possible consumer-based application is the correction ofvideo camcorder misfocus. Camcorder misfocus typically results from usererror, design defects such as a poorly designed zoom lens, or because anautofocus function is autoranging on the wrong part of the scene beingrecorded. Non-varying misfocus can be corrected for each image with thesame correction parameters. In the case of zoom lens misfocus, eachframe or portion of the video may require differing correctionparameters. In this application area, the invention can be incorporatedinto VCRs, camcorders, video editing systems, video processing machines,desktop video editing software, and video editing web sites, amongothers.

3. Another commercial application involves the correction of imagemisfocus experienced in remote video cameras utilizing digital signalprocessing. Particular examples include video conference cameras orsecurity cameras. In these scenarios, the video camera focus cannot beadequately or accessibly adjusted, and the video signal may in fact becompressed.

4. Video compression may involve motion compensation operations thatwere performed on the unfocused video image. Typical applicationsutilizing video compression include, for example, video conferencing,video mail, and web-based video-on-demand, to name a few. In theseparticular types of applications, the invention may be employed at thevideo receiver, or at some pre-processing stage prior to delivering thesignal to the video receiver. If the video compression introduces alimited number of artifacts, misfocus correction is accomplished aspresented herein. However, if the video compression introduces a highernumber of artifacts, the signal processing involved with the inventionmay greatly benefit from working closely with the video decompressionsignal processing. One particular implementation is where misfocuscorrections are first applied to a full video frame image. Then, forsome interval of time, misfocus correction is only applied to thechanging regions of the video image. A specific example may be wherelarge portions of a misfocused background can be corrected once, andthen reused in those same regions in subsequent video frames.

5. The misfocus correction techniques described herein are directlyapplicable to electron microscopy systems and applications. For example,electron microscope optics employ the wave properties of electrons tocreate a coherent optics environment that obeys the Fourier opticsstructures as coherent light (see, for example, John C. H. Spence,High-Resolution Electron Microscopy, third edition, 2003, Chapters 2–4,pp. 15–88). Electron beams found in electron microscopes have the samegeometric, optical physics characteristics generally found in coherentlight, and the same mathematical quadratic phase structure as indicatedin Levi [1] Section 19.2 for coherent light, which is the basis of thefractional Fourier transform in optical systems (see, for example, JohnC. H. Spence High-Resolution Electron Microscopy, third edition, 2003,Chapter 3, formula 3.9, pg. 55).

Misfocused Optical Path Phase Reconstruction

Most photographic and electronic image capture, storage, and productiontechnologies are only designed to operate with image amplitudeinformation, regardless as to whether the phase of the light is phasecoherent (as is the case with lasers) or phase noncoherent (as generallyfound in most light sources). In sharply focused images involvingnoncoherent light formed by classical geometric optics, this lack ofphase information is essentially of no consequence in many applications.

In representing the spatial distribution of light, the phase coefficientof the basis functions can be important; as an example, FIG. 3.6, p. 62of Digital Image Processing—Concepts, Algorithms, and ScientificApplications, by Bernd Jahne, Springer-Verlag, New York, 1991 [20] showsthe effect of loss and modification of basis function phase informationand the resulting distortion in the image. Note in this case the phaseinformation of the light in the original or reproduced image differsfrom the phase information applied to basis functions used forrepresenting the image.

In using fractional powers of the Fourier transform to represent opticaloperations, the fractional Fourier transform reorganizes the spatialdistribution of an image and the phase information as well. Here thebasis functions serve to represent the spatial distribution of light ina physical system and the phase of the complex coefficients multiplyingeach of the basis functions mathematically result from the fractionalFourier transform operation. In the calculation that leads to thefractional Fourier transform representation of a lens, complex-valuedcoefficients arise from the explicit accounting for phase shifts oflight that occurs as it travels through the optical lens (see Goodman[2], pages 77–96, and Levi [1], pages 779–784).

Thus, when correcting misfocused images using fractional powers of theFourier transform, the need may arise for the reconstruction of relativephase information that was lost by photographic and electronic imagecapture, storage, and production technologies that only capture andprocess image amplitude information.

In general, reconstruction of lost phase information has not previouslybeen accomplished with much success, but some embodiments of theinvention leverage specific properties of both the fractional Fouriertransform and an ideal correction condition. More specifically, what isprovided—for each given value of the focus correction parameter—is thecalculation of an associated reconstruction of the relative phaseinformation. Typically, the associated reconstruction will be inaccurateunless the given value of the focus correction parameter is one thatwill indeed correct the focus of the original misfocused image.

This particular aspect of the invention provides for the calculation ofan associated reconstruction of relative phase information by using thealgebraic group property of the fractional Fourier transform to backcalculate the lost relative phase conditions that would have existed, ifthat given specific focus correction setting resulted in a correctlyfocused image. For convergence of human or machine iterations towards anoptimal or near optimal focus correction, the system may also leveragethe continuity of variation of the phase reconstruction as the focuscorrection parameter is varied in the iterations.

To facilitate an understanding of the phase reconstruction aspect of theinvention, it is helpful to briefly summarize the some of the imagemisfocus correction aspects of the invention. This summary will be madewith reference to the various optical set-ups depicted in FIGS. 4–8, andis intended to provide observational details and examples of where andhow the relative phase reconstruction may be calculated (FIG. 9) andapplied (FIG. 10).

Misfocus Correction

FIG. 4 shows a general optical environment involving sources ofradiating light 400, a resulting original light organization(propagation direction, amplitude, and phase) 401 and its constituentphotons. Optical element 402 is shown performing an image-formingoptical operation, causing a modified light organization (propagationdirection, amplitude, and phase) 403 and its constituent photons,ultimately resulting in observed image 404. This figure shows that foreach light organization 401, 403 of light and photons, the propagationdirection, amplitude, and phase may be determined by a variety ofdifferent factors. For example, for a given propagation media,propagation direction, amplitude, and phase may be determined by suchthings as the separation distance between point light source 400 andoptical element 402, the pixel location in a transverse plane parallelto the direction of propagation, and light frequency/wavelength, amongothers.

FIG. 5 is an optical environment similar to that depicted in FIG. 4, butthe FIG. 5 environment includes only a single point light source 500. Inthis Figure, single point light source 500 includes exemplarypropagation rays 501 a, 501 b, 501 c that are presented to opticalelement 502. The optical element is shown imposing an optical operationon these rays, causing them to change direction 503 a, 503 b, 503 c.Each of the rays 503 a, 503 b, 503 c are shown spatially reconverging ata single point in the plane of image formation 504, which is a focusedimage plane.

FIG. 5 also shows directionally modified rays 503 a, 503 b, 503 cspatially diverging at short unfocused image plane 505 and longunfocused image plane 506, which are each transverse to the direction ofpropagation that is associated with images which are not in sharp focus,which will be referred to as nonfocused image planes. Furtherdescription of the optical environment shown in FIG. 5 will be presentedto expand on phase correction, and such description will be laterdiscussed with regard to FIGS. 9–10.

Reference is now made to FIGS. 6–8, which disclose techniques formathematical focus correction and provides a basis for understanding thephase correction aspect of the present invention. For clarity, the term“lens” will be used to refer to optical element 502, but the discussionapplies equally to other types of optical elements such as a system oflenses, graded-index material, and the like.

FIG. 6 provides an example of image information flow in accordance withsome embodiments of the present invention. As depicted in block 600, anoriginal misfocused image is adapted or converted as may be necessaryinto a digital file representation of light amplitude values 601.Examples of original misfocused images include natural or photographicimages. Digital file 601 may include compressed or uncompressed imageformats.

For a monochrome image, the light amplitude values are typicallyrepresented as scalar quantities, while color images typically involvevector quantities such as RBG values, YUV values, and the like. In someinstances, the digital file may have been subjected to file processessuch as compression, decompression, color model transformations, orother data modification processes to be rendered in the form of an arrayof light amplitude values 602. Monochrome images typically only includea single array of scalar values 602 a. In contrast, color images mayrequire one, two, or more additional arrays, such as arrays 602 b and602 c. A CMYB color model is a particular example of a multiple array,color image.

The array, or in some instances, arrays of light amplitude values 602may then be operated on by a fractional power of the Fourier transformoperation 603. This operation mathematically compensates for lensmisfocus causing the focus problems in the original misfocused image600. A result of this operation produces corrected array 604, and incase of a color model, exemplary subarrays 604 a, 604 b, 604 c resultfrom the separate application of the fractional power of the Fouriertransform operation 603 to exemplary subarrays 602 a, 602 b, 602 c. Ifdesired, each of the corrected subarrays 604 a, 604 b, 604 c may beconverted into a digital file representation of the corrected image 605;this digital file could be the same format, similar format, or anentirely different format from that of uncorrected, original digitalfile representation 601.

FIG. 7 shows an optical environment having nonfocused planes. Thisfigure shows that the power of the fractional Fourier transform operatorincreases as the separation distance between optical lens operation 502and image planes 504, 505 increases, up to a distance matching that ofthe lens law. In accordance with some aspects of the invention, and asexplained in Ludwig [5], Goodman [2], pages 77–96, and Levi [1], pages779–784, an exactly focused image corresponds to a fractional Fouriertransform power of exactly two. Furthermore, as previously described,misfocused image plane 505 lies short of the focused image plane 504,and corresponds to a fractional Fourier transform operation with a powerslightly less than two. The deviation in the power of the fractionalFourier transform operation corresponding to short misfocus image plane505 will be denoted (−∈_(s)), where the subscript “S” denotes “short.”Since an exactly focused image at focused image plane 504 corresponds toa fractional Fourier transform power of exactly two, this short misfocusmay be corrected by application of the fractional Fourier transformraised to the power (+ε_(s)), as indicated in block 701.

By mathematical extension, as described in [5], a long misfocused imageplane 506 that lies at a distance further away from optical element 502than does the focused image plane 504 would correspond to a fractionalFourier transform operation with a power slightly greater than two. Thedeviation in the power of the fractional Fourier transform operationcorresponding to long misfocus image plane 506 will be denoted (+μ_(L)),where the subscript “L” denotes “long.” This long misfocus may becorrected by application of the fractional Fourier transform raised tothe power (−ε_(L)), as indicated in block 702.

Relative Phase Information in the Misfocused Optical Path

In terms of geometric optics, misfocus present in short misfocused imageplane 505 and long misfocused image plane 506 generally correspond tonon-convergence of rays traced from point light source 500, throughoptical element 502, resulting in misfocused images planes 505 and 506.For example, FIGS. 5 and 7 show exemplary rays 501 a, 501 b, 501 cdiverging from point light source 500, passing through optical element502, and emerging as redirected rays 503 a, 503 b, 503 c. The redirectedrays are shown converging at a common point in focused image plane 504.However, it is important to note that these redirected rays converge atdiscreetly different points on misfocused image planes 505 and 506.

FIG. 8 is a more detailed view of image planes 504, 505 and 506. In thisfigure, rays 503 a, 503 b, 503 c are shown relative to focused imageplane 504, and misfocused image planes 505 and 506. This figure furthershows the path length differences that lead to phase shifts of thefocused and unfocused planes result from varying angles of incidence,denoted by θ₁ and θ₂. The distances of rays 503 a, 503 b, 503 c fromoptical element 502 are given by the following table:

TABLE 1 Distance to incidence Distance to Distance to incidence withshort misfocused incidence with with long misfocused Ray plane 505 focusplane 504 plane 506 503a δ₁ ^(S) δ₁ ^(F) δ₁ ^(L) 503b δ₀ ^(S) δ₀ ^(F) δ₀^(L) 503c δ₂ ^(S) δ₂ ^(F) δ₂ ^(L)Simple geometry yields the following inequality relationships:δ₀ ^(S)<δ₀ ^(F)<δ₀ ^(L)  (21)δ₁ ^(S)<δ₁ ^(F)<δ₁ ^(L)  (22)δ₂ ^(S)<δ₂ ^(F)<δ₂ ^(L)  (23)For a given wavelength λ, the phase shift ψ created by adistance-of-travel variation δ is given by the following:ψ=2πδ/λ  (24)so the variation in separation distance between the focus image plane504 and the misfocus image planes 505, 506 is seen to introduce phaseshifts along each ray.

Further, for π/2>θ₁>θ₂>0, as is the case shown in FIG. 8, simpletrigonometry gives:δ₀ ^(F)=δ₁ ^(F)sin θ₁  (25)δ₀ ^(F)=δ₂ ^(F)sin θ₂  (26)1>sin θ₁>sin θ₂>0  (27)which in turn yields the inequality relationships:δ₀ ^(S)<δ₂ ^(S)<δ₁ ^(S)  (28)δ₀ ^(F)<δ₂ ^(F)<δ₁ ^(F)  (29)δ₀ ^(L)<δ₂ ^(L)<δ₁ ^(L)  (30)Again, for a given wavelength λ, the phase shift ψ created by adistance-of-travel variation δ is given by the following:ψ=2πδ/λ  (31)so the variation in separation distance between focused image plane 504and the misfocused image planes 505, 506 is seen to introducenon-uniform phase shifts along each ray. Thus the misfocus of theoriginal optical path involved in creating the original image (forexample, 600 in FIG. 6) introduces a non-uniform phase shift across therays of various incident angles, and this phase shift varies with thedistance of separation between the positions of misfocused image planes505, 506, and the focused image plane 504.

Referring again to FIGS. 6 and 7, an example of how a misfocused image600 may be corrected will now be described. A misfocused image requiringcorrection will originate either from short misfocused plane 505 or longmisfocused plane 506. In situations where misfocused image 600originates from short misfocused plane 505, misfocus correction may beobtained by applying a fractional Fourier transform operation raised tothe power (+ε_(s)), as indicated in block 701. On the other hand, insituations where misfocused image 600 originates from long misfocusedplane 506, misfocus correction may be obtained by applying a fractionalFourier transform operation raised to the power (−ε_(L)), as indicatedin block 702.

In general, the fractional Fourier transform operation creates resultsthat are complex-valued. In the case of the discrete fractional Fouriertransform operation, as used herein, this operation may be implementedas, or is equivalent to, a generalized, complex-valued arraymultiplication on the array image of light amplitudes (e.g., φ). In thesignal domain, complex-valued multiplication of a light amplitude arrayelement, υ_(ij), by a complex-valued operator element φ, results in anamplitude scaling corresponding to the polar or phasor amplitude of φ,and a phase shift corresponding to the polar or phasor phase of φ.

FIG. 9 shows a series of formulas that may be used in accordance withthe present invention. As indicated in block 901, the fractional Fouriertransform operation array (FrFT) is symbolically represented as theproduct of an amplitude information array component and a phaseinformation array component. The remaining portions of FIG. 9 illustrateone technique for computing phase correction in conjunction with thecorrection of image misfocus. For example, block 902 shows one approachfor correcting image misfocus, but this particular technique does notprovide for phase correction. Image correction may proceed by firstnoting that the misfocused optics corresponds to a fractional Fouriertransform of power 2−ε for some unknown value of ε; here ε may bepositive (short-misfocus) or negative (long-misfocus). Next, thefractional Fourier transform may be mathematically applied with various,systematically selected trial powers until a suitable trial power isfound. A particular example may be where the trial power is effectivelyequal to the unknown value of ε. The resulting mathematically correctedimage appears in focus and a corrected image is thus produced.

Referring still to FIG. 9, block 903 depicts the misfocus correctiontechnique of block 901, as applied to the approach shown in block 902.This particular technique accounts for the amplitude and phasecomponents of the optical and mathematical fractional Fourier transformoperations. In particular, there is an amplitude and phase for themisfocused optics, which led to the original misfocused image 600.

As previously noted, conventional photographic and electronic imagecapture, storage, and production technologies typically only process oruse image amplitude information, and were phase information is notrequired or desired. In these types of systems, the relative phaseinformation created within the original misfocused optical path is lostsince amplitude information is the only image information that isconveyed. This particular scenario is depicted in block 904, which showsoriginal misfocused image 600 undergoing misfocus correction, eventhought its relative phase information is not available. In allapplicable cases relevant to the present invention (for example, 0<ε<2),the phase information is not uniformly zero phase, and thus the missingphase information gives an inaccurate result (that is, not equal to thefocused case of the Fourier transform raised to the power 2) for whatshould have been the effective correction.

Relative Phase Restoration

In accordance with some embodiments, missing phase information may bereintroduced by absorbing it within the math correction stage, as shownblock 905. This absorbing technique results in a phase-restored mathcorrection of the form:Φ(F^(2−γ))F^(γ)  (32)where the following symbolic notation is used:Φ(X)=phase(X)  (33)

In the case where γ is close enough to be effectively equal to ε, thephase correction will effectively be equal to the value necessary torestore the lost relative phase information. Note that this expressiondepends only on γ, and thus phase correction may be obtained bysystematically iterating γ towards the unknown value of ε, which isassociated with the misfocused image. Thus the iteration, computation,manual adjustment, and automatic optimization systems, methods, andstrategies of non-phase applications of image misfocus correction may beapplied in essentially the same fashion as the phase correctingapplications of image misfocus correction by simply substituting F^(γ)with Φ(F^(2−γ))F^(γ) in iterations or manual adjustments.

FIG. 10 provides an example of image information flow in accordance withsome embodiments of the invention. This embodiment is similar to FIG. 6is many respects, but the technique shown in FIG. 10 further includesphase restoration component 1001 coupled with focus correction component603. In operation, image array 602 is passed to phase restorationcomponent 1001, which pre-operates on image array 602. After thepre-operation calculation have been performed, fractional Fouriertransform operation 603 is applied to the image array.

Numerical Calculation of Relative Phase Restoration

Next, the calculation of the phase-restored mathematical correction isconsidered. Leveraging two-group antislavery properties of thefractional Fourier transform operation, the additional computation canbe made relatively small.

In the original eigenfunction/eigenvector series definitions for boththe continuous and discrete forms of the fractional Fourier transform ofpower α, the nth eigenfunction/eigenvectors are multiplied by:e^(−inπα/2)  (34)Using this equation and replacing α with (2−γ) gives:

$\begin{matrix}\begin{matrix}{{\mathbb{e}}^{{- {\mathbb{i}}}\; n\;{{\pi{({2 - \gamma})}}/2}} = {{\mathbb{e}}^{{- {\mathbb{i}}}\; n\;\pi}{\mathbb{e}}^{{- {\mathbb{i}}}\; n\;{{\pi{({- \gamma})}}/2}}}} \\{= {\left( {- 1} \right)^{n}{\mathbb{e}}^{{- {\mathbb{i}}}\; n\;{{\pi{({- \gamma})}}/2}}}}\end{matrix} & (35)\end{matrix}$for both the continuous and discrete forms of the fractional Fouriertransform. Note that the following equation:e^(−inπ(−γ))  (36)can be rewritten as:e ^(−inπ(−γ)) =e ^(inπγ)=(e ^(−inπγ))*  (37)where (X)* denotes the complex conjugate of X.

Also, because the nth Hermite function h_(n)(y) is odd in y for odd n,and even in y for even n, such that:h _(n)(−y)=(−1)^(n) h _(n)(−y)  (38)so that in the series definition the nth term behaves as:

$\begin{matrix}\begin{matrix}{{{h_{n}(x)}{h_{n}(y)}{\mathbb{e}}^{{- {\mathbb{i}}}\; n\;{{\pi{({2 - \gamma})}}/2}}} = {{h_{n}(x)}{h_{n}(y)}\left( {- 1} \right)^{n}{\mathbb{e}}^{{- {\mathbb{i}}}\; n\;{{\pi{({- \gamma})}}/2}}}} \\{= {{h_{n}(x)}{h_{n}\left( {- y} \right)}{\mathbb{e}}^{{- {\mathbb{i}}}\; n\;{{\pi{({- \gamma})}}/2}}}} \\{= {{h_{n}(x)}{h_{n}\left( {- y} \right)}\left( {\mathbb{e}}^{{- {\mathbb{i}}}\; n\;\pi\;\gamma} \right)^{*}}}\end{matrix} & (39)\end{matrix}$For both the continuous and discrete forms of the fractional Fouriertransform, replacing h_(n)(y) with h_(n)(−y) is equivalent to reversing,or taking the mirror image, of h_(n)(y). In particular, for the discreteform of the fractional Fourier transform, this amounts to reversing theorder of terms in the eigenvectors coming out of the similaritytransformation, and because of the even-symmetry/odd-antisymmetry of theHermite functions and the fractional Fourier transform discreteeigenvectors, this need only be done for the odd number eigenvectors.

Further, since the Hermite functions and discrete Fourier transformeigenvectors are real-valued, the complex conjugate can be taken on theentire term, not just the exponential, as shown by:h _(n)(x)h _(n)(−y)(e ^(−inπy))*=[h _(n)(x)h _(n)(−y)e ^(−inπγ)]*  (40)Since complex conjugation commutes with addition, all these series termscan be calculated and summed completely before complex conjugation, andthen one complex conjugation can be applied to the sum, resulting in thesame outcome.

The relative phase-restored mathematical correction can thus becalculated directly, for example, by the following exemplary algorithmor its mathematical or logistic equivalents:

-   1. For a given value of γ, compute F^(γ) using the Fourier transform    eigenvectors in an ordered similarity transformation matrix;-   2. For the odd-indexed eigenvectors, either reverse the order or the    sign of its terms to get a modified similarity transformation;-   3. Compute the complete resulting matrix calculations as would be    done to obtain a fractional Fourier transform, but using this    modified similarity transformation;-   4. Calculate the complex conjugate of the result of operation (3) to    get the phase restoration, (Φ(F^(γ)))*; and-   5. Calculate the array product of the operation (1) and    operation (4) to form the phase-restored focus correction    (Φ(F^(γ)))*F^(γ).

As an example of a mathematical or logistic equivalent to the justdescribed series of operations, note the commonality of the calculationsin operations (1) and (3), differing only in how the odd-indexedeigenvectors are handled in the calculation, and in one version, only bya sign change. An example of a mathematical or logistic equivalent tothe above exemplary technique would be:

-   1. For a given value of γ, partially compute F^(γ) using only the    even-indexed Fourier transform eigenvectors;-   2. Next, partially compute the remainder of F^(γ) using only the    odd-indexed Fourier transform eigenvectors;-   3. Add the results of operation (1) and (2) to get F^(γ)-   4. Subtract the result of operation (2) from the result of    operation (1) to obtain a portion of the phase restoration;-   5. Calculate the complex conjugate of the result of operation (4) to    obtain the phase restoration (Φ(F^(γ)))*; and-   6. Calculate the array product of operations (1) and (4) to form    (Φ(F^(γ)))*F^(γ).

In many situations, partially computing two parts of one similaritytransformation, as described in the second exemplary algorithm, could befar more efficient than performing two full similarity transformationcalculations, as described in the first exemplary algorithm. One skilledin the art will recognize many possible variations with differingadvantages, and that these advantages may also vary with differingcomputational architectures and processor languages.

Embedding Phase Restoration within Image Misfocus Correction

Where relative phase-restoration is required or desired in mathematicalfocus correction using the fractional Fourier transform, phaserestoration element 1001 may be used in combination with focuscorrection element 603, as depicted in FIG. 10.

It is to be realized that in image misfocus correction applicationswhich do not account for phase restoration, pre-computed values of F^(γ)may be stored, fetched, and multiplied as needed or desired. Similarly,in image misfocus correction applications which provide for phaserestoration, pre-computed values of Φ(F^(γ)))*F^(γ) may also be stored,fetched, and multiplied as needed or desired. For example, pre-computedvalues of phase reconstructions may be stored corresponding to powers ofthe fractional Fourier transform, such that the powers are related byroots of the number 2, or realized in correspondence to binaryrepresentations of fractions, or both. In these compositions, care mayneed to be taken since the array multiplications may not freely commutedue to the nonlinear phase extraction steps.

Each of the various techniques for computing the phase-restored focuscorrection may include differing methods for implementing pre-computedphase-restorations. For example, in comparing the first and secondexemplary algorithms, predominated values may be made and stored for anyof:

-   -   First example algorithm operation (5) or its equivalent second        example algorithm operation (6);    -   First example algorithm operation (4) or its equivalent second        example algorithm operation (5); and    -   Second example algorithm operations (1) and (2) with additional        completing computations provided as needed.

Again, it is noted that these phase restoration techniques can apply toany situation involving fractional Fourier transform optics, includingelectron microscopy processes and the global or localized correction ofmisfocus from electron microscopy images lacking phase information.Localized phase-restored misfocus correction using the techniquesdisclosed herein may be particularly useful in three-dimensional,electron microscopy and tomography where a wide field is involved in atleast one dimension of imaging.

While the invention has been described in detail with reference todisclosed embodiments, various modifications within the scope of theinvention will be apparent to those of ordinary skill in thistechnological field. It is to be appreciated that features describedwith respect to one embodiment typically may be applied to otherembodiments. Therefore, the invention properly is to be construed withreference to the claims.

REFERENCES CITED

The following references are cited herein:

-   [1] L. Levi, Applied Optics, Volume 2 (Section 19.2), Wiley, New    York, 1980;-   [2] J. W. Goodman, Introduction to Fourier Optics, McGraw-Hill, New    York, 1968;-   [3] K. Iizuka, Engineering Optics, Second Edition, Springer-Verlag,    1987;-   [4] A. Papoulis, Systems and Transforms with Applications in Optics,    Krieger, Malabar, Florida, 1986;-   [5] L. F. Ludwig, “General Thin-Lens Action on Spatial Intensity    (Amplitude) Distribution Behaves as Non-Integer Powers of Fourier    Transform,” Spatial Light Modulators and Applications Conference,    South Lake Tahoe, 1988;-   [6] R. Dorsch, “Fractional Fourier Transformer of Variable Order    Based on a Modular Lens System,” in Applied Optics, vol. 34, no. 26,    pp. 6016–6020, September 1995;-   [7] E. U. Condon, “Immersion of the Fourier Transform in a    Continuous Group of Functional Transforms,” in Proceedings of the    National Academy of Science, vol. 23, pp. 158–161, 1937;-   [8] V. Bargmann, “On a Hilbert Space of Analytical Functions and an    Associated Integral Transform,” Comm. Pure Appl. Math, Volume 14,    1961, 187–214;-   [9] V. Namias, “The Fractional Order Fourier Transform and its    Application to Quantum Mechanics,” in J. of Institute of Mathematics    and Applications, vol. 25, pp. 241–265, 1980;-   [10] B. W. Dickinson and D. Steiglitz, “Eigenvectors and Functions    of the Discrete Fourier Transform,” in IEEE Transactions on    Acoustics, Speech, and Signal Processing, vol. ASSP-30, no. 1,    February 1982;-   [11] F. H. Kerr, “A Distributional Approach to Namias' Fractional    Fourier Transforms,” in Proceedings of the Royal Society of    Edinburgh, vol. 108A, pp. 133–143, 1983;-   [12] F. H. Kerr, “On Namias' Fractional Fourier Transforms,” in    IMA J. of Applied Mathematics, vol. 39, no. 2, pp. 159–175, 1987;-   [13] P. J. Davis, Interpolation and Approximation, Dover, New York,    1975;-   [14] N. I. Achieser, Theory of Approximation, Dover, New York, 1992;-   [15] G. B. Folland, Harmonic Analysis in Phase Space, Princeton    University Press, Princeton, N.J., 1989;-   [16] N. N. Lebedev, Special Functions and their Applications, Dover,    New York, 1965;-   [17] N. Wiener, The Fourier Integral and Certain of Its    Applications, (Dover Publications, Inc., New York, 1958) originally    Cambridge University Press, Cambridge, England, 1933;-   [18] S. Thangavelu, Lectures on Hermite and Laguerre Expansions,    Princeton University Press, Princeton, N.J., 1993;-   [19] “Taking the Fuzz out of Photos,” Newsweek, Volume CXV, Number    2, Jan. 8, 1990; and-   [20] Jahne, Bernd, Digital Image Processing—Concepts, Algorithms,    and Scientific Applications, Springer-Verlag, N.Y., 1991.

1. A method for reconstructing relative phase information in aniterative misfocused image correction process, said method comprising:(a) receiving misfocused image data from an image source, said imagedata comprising image amplitude information and lacking relative phaseinformation; (b) generating a trial power value for a fractional Fouriertransform correction operation according to an optimization algorithm;(c) determining a trial fractional Fourier transform correctionoperation based upon said trial power value; (d) determining trial phaserestoration information based upon said trial power value; ( e) applyingsaid trial phase restoration information and said trial fractionalFourier transform correction operation on said image data to correct alevel of misfocus of said image data; and (f) determining if a desiredlevel of correction of said misfocus of said image data has beenachieved, wherein if said desired level has not been achieved, then anew value for said power value of said fractional Fourier transformcorrection operation is generated according to said optimizationalgorithm and said operations (c) through (f) are repeated.
 2. Themethod according to claim 1, wherein said trial phase restorationinformation is determined by the difference between said trial powervalue and an in-focus power of said fractional Fourier transformcorrection operation.
 3. The method according to claim 2, wherein saidin-focus power is a power that causes a fractional Fourier transformraised to said in-focus power to be equivalent to a Fourier transformcomposed with itself.
 4. The method according to claim 1, wherein saidtrial fractional Fourier transform correction operation is determined byat least one pre-computed fractional Fourier transform operation.
 5. Themethod according to claim 1, wherein said trial fractional Fouriertransform correction operation is determined using a plurality ofpre-computed fractional Fourier transform operations.
 6. The methodaccording to claim 1, wherein said trial fractional Fourier transformcorrection operation is determined by a direct calculation from adefinition parameterized by said trial power value.
 7. The methodaccording to claim 1, wherein said trial phase restoration informationis determined by an operation using at least one pre-computed phaserestoration value.
 8. The method according to claim 1, wherein saidtrial phase restoration information is determined by an operation usinga plurality of pre-computed phase restoration values.
 9. The methodaccording to claim 1, wherein said trial phase restoration informationis determined by a direct calculation from a definition parameterized bysaid trial power value.
 10. The method according to claim 2, whereinsaid trial fractional Fourier transform correction operation is ofperiod four and said in-focus power is two.
 11. The method according toclaim 1, wherein said image source only conveys amplitude information.12. The method according to claim 1, wherein said misfocused image datais selected from the group consisting of an electronic signal, datafile, photography paper, photographic slide, motion photography,decompressed digital image, video frame, video stills, and motion video.13. The method according to claim 1, wherein said image source renderssaid image data from an electronic signal source.
 14. The methodaccording to claim 1, wherein said image source renders said image datafrom a photographic media source.
 15. The method according to claim 1,wherein said misfocused image data comprises motion video.
 16. Themethod according to claim 1, wherein said misfocused image datacomprises a data file.
 17. The method according to claim 1, wherein saidmisfocused image data comprises motion photography.
 18. The methodaccording to claim 1, wherein said misfocused image data has been storedand retrieved.
 19. The method according to claim 1, wherein saiddetermining operation (f) is manually performed by a human operator. 20.The method according to claim 1, wherein said determining operation (f)is performed by an automatic process.
 21. The method according to claim1, wherein said trial phase restoration information is determined byperforming mathematical operations on said trial fractional Fouriertransform correction operation.
 22. The method according to claim 1,wherein said trial phase restoration information is determined byreconfiguring calculations used to determine said trial fractionalFourier transform correction operation.
 23. The method according toclaim 1, wherein said trial phase restoration information is determinedby calculations involving at least one partial result that is also usedto determine said trial fractional Fourier transform correctionoperation obtained in operation (c).
 24. The method according to claim1, wherein said trial fractional Fourier transform correction operationis determined by composing a plurality of pre-computed fractionalFourier transform operations, wherein each of said plurality of saidpre-computed fractional Fourier transform correction operationscorrespond to associated power values that are selected binaryfractions.
 25. The method according to claim 24, said method furthercomprising: selectively choosing which transform operations of saidplurality of pre-computed fractional Fourier transform operations are tobe used in said composing, wherein said choosing is determined by thevalue of digits in a binary fraction representation of said trial powervalue generated in operation (b).
 26. The method according to claim 1,wherein said trial phase restoration information is determined bycomposing a plurality of pre-computed phase restoration values, whereinsaid plurality of said pre-computed phase restoration values correspondto associated power values that are selected binary fractions.
 27. Themethod according to claim 26, said method further comprising:selectively choosing which transform operations of said plurality ofpre-computed fractional Fourier transform operations are to be used insaid composing, wherein said choosing is determined by the value ofdigits in a binary fraction representation of said trial power valuegenerated in operation (b).
 28. The method according to claim 1, whereinsaid image data is obtained from an electron microscopy device.
 29. Themethod according to claim 28, wherein said trial phase restorationinformation and said trial fractional Fourier transform correctionoperation are applied on a selected portion of said image data tocorrect a level of said misfocus at said selected portion of said imagedata.
 30. The method according to claim 1, wherein said trial phaserestoration information and said trial fractional Fourier transformcorrection operation are applied on a selected portion of said imagedata to correct a level of said misfocus at said selected portion ofsaid image data.
 31. The method according to claim 1, said methodfurther comprising: providing said corrected image data to an imageobservation element.
 32. A system for reconstructing relative phaseinformation in an iterative misfocused image correction process, saidsystem comprising: an image source providing misfocused image data, saidmisfocused image data comprising image amplitude information and lackingrelative phase information; at least one processor adapted to providesaid iterative misfocused image correction process according to a methodcomprising: (a) generating a trial power value for a fractional Fouriertransform correction operation according to an optimization algorithm;(b) determining a trial fractional Fourier transform correctionoperation based upon said trial power value; (c) determining trial phaserestoration information based upon said trial power value; (d) applyingsaid trial phase restoration information and said trial fractionalFourier transform correction operation on said image data to correct alevel of misfocus of said image data; and (e) determining if a desiredlevel of correction of said misfocus of said image data has beenachieved, wherein if said desired level has not been achieved, then anew value for said power value of said fractional Fourier transformcorrection operation is generated according to said optimizationalgorithm and said operations (b) through (e) are repeated.
 33. Thesystem according to claim 32, wherein said trial phase restorationinformation is determined by a direct calculation from a definitionparameterized by said trial power value.
 34. The system according toclaim 32, wherein said image source only conveys amplitude information.35. The system according to claim 32, wherein said misfocused image datais selected from the group consisting of an electronic signal, datafile, photography paper, photographic slide, motion photography,decompressed digital image, video frame, video stills, and motion video.36. The system according to claim 32, wherein said misfocused image datacomprises motion video.
 37. The system according to claim 32, whereinsaid misfocused image data comprises motion photography.
 38. The systemaccording to claim 32, wherein said trial phase restoration informationis determined by performing mathematical operations on said trialfractional Fourier transform correction operation.
 39. The systemaccording to claim 32, wherein said trial phase restoration informationis determined by reconfiguring calculations used to determine said trialfractional Fourier transform correction operation.
 40. The systemaccording to claim 32, wherein said trial phase restoration informationand said trial fractional Fourier transform correction operation areapplied on a selected portion of said image data to correct a level ofsaid misfocus at said selected portion of said image data.
 41. A methodfor reconstructing relative phase information in a misfocused imagecorrection process, said method comprising: (a) receiving misfocusedimage data from an image source, said image data comprising imageamplitude information and lacking relative phase information; (b)generating a power value for a fractional Fourier transform correctionoperation according to a power calculation algorithm; (c) determining afractional Fourier transform correction operation based upon said powervalue; (d) determining phase restoration information based upon saidpower value; and (e) applying said phase restoration information andsaid fractional Fourier transform correction operation on said imagedata to correct a level of misfocus of said image data.
 42. The methodaccording to claim 41, said method further comprising: repeatingoperations (b) through (e) until a desired level of correction of saidmisfocus of said image data has been achieved.
 43. The method accordingto claim 41, wherein said power calculation algorithm generates saidpower value according to an optimization algorithm.
 44. The methodaccording to claim 41, wherein said phase restoration information isdetermined by a direct calculation from a definition parameterized bysaid power value.
 45. The method according to claim 41, wherein saidimage source only conveys amplitude information.
 46. The methodaccording to claim 41, wherein said misfocused image data is selectedfrom the group consisting of an electronic signal, data file,photography paper, photographic slide, motion photography, decompresseddigital image, video frame, video stills, and motion video.
 47. Themethod according to claim 41, wherein said misfocused image datacomprises motion video.
 48. The method according to claim 41, whereinsaid misfocused image data comprises motion photography.
 49. The methodaccording to claim 41, wherein said phase restoration information isdetermined by performing mathematical operations on said fractionalFourier transform correction operation.
 50. The method according toclaim 41, wherein said phase restoration information is determined byreconfiguring calculations used to determine said fractional Fouriertransform correction operation.
 51. The method according to claim 41,wherein said phase restoration information and said fractional Fouriertransform correction operation are applied on a selected portion of saidimage data to correct a level of said misfocus at said selected portionof said image data.
 52. A system for reconstructing relative phaseinformation in a misfocused image correction process, said systemcomprising: an image source providing misfocused image data, saidmisfocused image data comprising image amplitude information and lackingrelative phase information; at least one processor adapted to providesaid misfocused image correction process according to a methodcomprising: (a) generating a power value for a fractional Fouriertransform correction operation according to a power calculationalgorithm; (b) determining a fractional Fourier transform correctionoperation based upon said power value; (c) determining phase restorationinformation based upon said power value; and (d) applying said phaserestoration information and said fractional Fourier transform correctionoperation on said image data to correct a level of misfocus of saidimage data.
 53. The system according to claim 52, wherein operations (a)through (d) are repeated until a desired level of correction of saidmisfocus of said image data has been achieved.
 54. The system accordingto claim 52, wherein said power calculation algorithm generates saidpower value according to an optimization algorithm.
 55. The systemaccording to claim 52, wherein said phase restoration information isdetermined by a direct calculation from a definition parameterized bysaid power value.
 56. The system according to claim 52, wherein saidimage source only conveys amplitude information.
 57. The systemaccording to claim 52, wherein said misfocused image data is selectedfrom the group consisting of an electronic signal, data file,photography paper, photographic slide, motion photography, decompresseddigital image, video frame, video stills, and motion video.
 58. Thesystem according to claim 52, wherein said misfocused image datacomprises motion video.
 59. The system according to claim 52, whereinsaid misfocused image data comprises motion photography.
 60. The systemaccording to claim 52, wherein said phase restoration information isdetermined by performing mathematical operations on said fractionalFourier transform correction operation.
 61. The system according toclaim 52, wherein said phase restoration information is determined byreconfiguring calculations used to determine said fractional Fouriertransform correction operation.
 62. The system according to claim 52,wherein said phase restoration information and said fractional Fouriertransform correction operation are applied on a selected portion of saidimage data to correct a level of said misfocus at said selected portionof said image data.